CobWeb - a Handwriting Recognition Based Writing Environment for Children

This paper describes work that has investigated the usability of handwriting recognition technology for writing with children. The results show that for the age group considered, writing at the tablet was quite efficient, and children were able to write reasonable stories. Some of the key usability problems with the handwriting recognition technology are described and a discussion of possible future trends in digital writing is presented. Overview Introduction The Writing Process In the UK classroom, children spend between 30% and 60% of their school classroom time doing writing activities (McHale & Cermak, 1992). The freedom afforded to the child during the writing activity ranges from the rote copying of text to completely free, creative, text production. Most writing falls between these extremes, and involves composition work in response to some stimulus. The early parts of this process are described in Figure 1, which outlines the multiple processes that children have to deal with to produce written text. Figure 1 The Process of Composition In this diagram, the writing process is broken into three stages: The first, A, is largely psychological, whereas B and C are more concerned with the physical processes of writing. These latter two stages are labelled separately as if the ‘writing’ were to only exist in an audio format (and this is an interesting option that is not further explored in this paper) process B would not be required. During any of these three stages, the child may encounter difficulties. Spelling and punctuation are particularly difficult and in order for children to eventually have a ‘nice’ piece of work multiple revisions are sometimes required. These revisions include spelling changes, the insertion of missed words, and sometimes, but less commonly, rearrangements of phrases. It is usual for children to limit themselves to revisions that will not ‘mess up’ their work too much. As the revision process is difficult, it can sometimes happen that a child begins a correction, only to forget what he was going to write by the time he has finished with the eraser (Swanson & Berninger, 1996). Using Computers to Support the Writing Process There is widespread support in the education system for the use of electronic tools to assist children in writing. Researchers at Newcastle have identified three benefits that computers and their associated technology can bring to literacy (Moseley et al., 1999); ♦ The capacity to present or represent ideas dynamically or in multiple forms ♦ The facility for providing feedback to pupils as they are working ♦ The capacity to present information in easily changed forms Software that is commonly used to support writing in the classroom is generally either instructional (educational) or functional (enabling). Spelling instruction and grammar testing software fall into the first category whilst web development tools, desktop publishing and word processors fall into the second. One feature that is shared amongst the functional applications is that the work that is produced can be made to look good. This can motivate young writers but can also result in children becoming more concerned about the visual attractiveness of their text than the quality of the writing that it represents (Day, 1994). The single most used functional software product for writing is the word processor. There are many word processors that are designed especially for children and these have different means to help children accomplish the three stages described in Figure 1. Different software (Figure 2 and Figure 3) supports the planning phase of the writing process with the opportunity for children to arrange their plots and make notes about what will be placed in each section before they start to compose. Figure 2 Clicker® Software Figure 3 Planning Pages from CoWriter® A popular writing package in English Schools is WriteOn® for Windows. This can be found in many infant classrooms and it incorporates a word bank, attractive fonts and a colourful interface. The interface for the second level (there are four levels so that the software can be customised with respect to the age of the child) of this package is shown in Figure 4. The word bank simplifies the text input process as the child does not need to type out the words but can simply select them using a mouse. Figure 4 Write On® for Windows A further example of this approach to text input is shown in Figure 5 that shows the word bank from Clicker® Software. Figure 5 Clicker Word Processing Software How useful is Word Processing Software for beginning writers? There has been considerable research on the use of word processing in the primary classroom, however some of this research is contradictory. Some of the studies have only small cohorts and the additional features that certain word processors offer can affect reported results; one such example is work by (Beck & Fetherston, 2003) that made claims for the effectiveness of word processing but had only seven children and the word processing included many features including drawing and story starters. Research has suggested that word processors can be useful tools for children learning to write (Cochran-Smith, 1991), (Zbikowski & Pan, 1995). A study by (Jones & Pellegrini, 1996) concluded that children wrote more cohesively at the word processor than when using pen and paper; observational work by (Fisher, 1994) showed that children could compose work together at the computer and during this, their talk was task focused and cooperative. Papert (1980) suggests that a computer with word processing software affords the child the luxury of being able to revise and rework their ideas, and therefore becomes a powerful intellectual product. However, this argument is countered by research findings that suggest that with young children; revision of work is unusual with inexperienced writers generally only revising their work if prompted and even then, they will often only make word changes, rather than carrying out substantial rewrites (Monahan, 1984), (Hult, 1986). Using the Keyboard to enter text For children to be able to use the features that text or word processing allows on a computer, they have to first get their work into the computer. This can be done indirectly, by using scanning technology, or directly by the child, or some other person. Direct text entry is usually carried out using the alphabetic keyboard and the process that is associated with this is generally referred to as typing or keyboarding (this is less often used as it confuses the actions on an alphabetic keyboard with those on a musical keyboard). Alphabetic keyboards can be presented in different ways; the two most common are the QWERTY keyboard and the Dvorak keyboard. There is a range of specialist keyboards, known as concept keyboards, which have been specially designed for use with young children and with disabled adults. These keyboards are generally non-alphabetic, using pictures to depict actions or words. The process of typing can be broken into five phases, these are, character recognition, storage, motor activity, keystroke and feedback (Cooper, 1983). Each of these phases presents difficulties for children. Character recognition is the process of recognising the letter on the keyboard, sometimes difficult for children due to the tendency to use upper case representations. Storage refers to the ability of the typist to be able to read ahead – possibly four to eight characters at a time; this is reduced for children who do not have the same ability and knowledge with the keyboard. The motor activity is the movement of the fingers to the keys, which will be slower for children as their hands are smaller and therefore they have to make larger gross motor movements than adults. The keystroke is the pressure needed to press the key and feedback refers to the realisation that the correct character has been inputted which is essential for error detection and correction. It is possible to become quite skilled at the keyboard; but many people, and particularly children find typing difficult (Norman & Fisher, 1982). Recent advances in technology allow handwritten text to be input into a computer where it is then turned into digitized, ASCII text, such as that that is manipulated in word processing software. This is the technology that is explored in the remainder of the paper. Handwriting for Text Input Development of Handwriting Handwriting is the term used to describe the formation of the letters or words of a language by an individual. Handwriting competence is measured by the legibility of the resulting work and the attainment of a reasonable speed. People are often judged by their handwriting, and it is not unusual for a child with poor handwriting to develop a poor self-image (Sassoon, 1990). Handwriting may be ‘printed’ (discrete) or ‘joined up’ (cursive). It has been traditionally the case that English school children learnt to write using discrete writing and then moved onto cursive writing at aged around eight or nine. There is considerable support in the writing community for a change to this pattern, with many writers suggesting that children should use cursive script at a much younger age (Nicholson, 1999), (Bearne, 1998), (Peters, 1985). (Peters, 1985) view is that children should learn cursive writing at Key Stage 1; believing that ♦ It will result in children understanding the concept of a word at an earlier age ♦ It will also result in better letter formation ♦ Children don’t have to ‘adapt’ at around eight ♦ The movements involved in cursive writing will produce better spelling. This last claim is particularly interesting; Peters claims that ‘Quality of handwriting is highly correlated with spelling attainment’ and this relationship is supported by (Bearne, 1998) who attributes the connection between handwriting and spelling to kinaesthetic memory. For a child to learn handwriting, he has to learn how to form the individual letters correctly (many with anti-clockwise curves), he has to realise that the direction of the letter is important (as in b and d), that ascenders and descenders go above and below the line and then later, he has to learn to make joins. The ability to orient and form letters correctly is actually very difficult to measure, and may be one reason why so many children acquire poor handwriting habits. Handwriting Recognition Handwriting (or written language) recognition is the task of transforming language represented in its spatial form of graphical marks into its symbolic representation. (It is sometimes referred to as Intelligent Character Recognition (ICR) – this definition sets it apart from Optical Character Recognition (OCR), which is the act of recognising printed rather than handwritten images). For English orthography this symbolic representation is as characters, typically represented as ASCII. The transformation is therefore a two-stage process that involves the digitisation of the writing and then the assignment of the digitised characters to ASCII text. The symbolic classes that are generally used include the upper and lower case characters, the ten digits, and special symbols, for instance punctuation marks (Plamondon & Srihari, 2000). The methods used to carry out handwriting recognition depend on many parameters. Some of the features that distinguish handwriting recognition systems are summarised in Table 1 and are expanded on in the following section: Writing parameters Range Capture Offline vs. Online Recognition mode Real time vs. Batch Writing mode Discrete characters vs. Cursive Language model Character or word level recognition to context sensitive using language models Vocabulary Small < 20 (constrained) to large >20,000 words (unconstrained) Writing style Copied vs. Spontaneous writing Enrolment Writer dependent vs. Writer Independent (omni – handwriting) Table 1 Classification of handwriting recognition systems Capture of the handwritten writing can be done in two ways; writing that exists on paper is optically scanned resulting in a bitmap of each character; text that is captured onto an electrically sensitive surface, such as a graphics tablet or a tablet PC, is available as a time sequence of pen-co-ordinates. An example of a graphics tablet and pen is shown in Figure 6. Figure 6 A Wacom Tablet and Pen These two different modes of digitisation are referred to as online and offline recognition respectively. With online recognition, the writing can subsequently be recognised in real time, as each word is written, or as a batch process after the writing is completed (Srihari & Srihari, 1997). The handwriting that is input to the process may be cursive (joined up) or discrete (printed). It is common for writers to use a combination of discrete and cursive text and so most recognition software needs to be able to deal with both individual letters and letter joins. Writing typically includes flourishes, pen marks and badly formed words and so in addition to initial stroke matching (which is able to identify many single characters), many recognisers use dictionaries (allowing them to apply word level recognition) and language models (sentence level recognition) to recover strings of characters and words from writing that has flourishes and missing bits (Plamondon & Srihari, 2000). Text entry to a system may be constrained or unconstrained. In a constrained system, the user or the alphabet (or both) is subject to controls. Examples of such controls include limiting the character set (perhaps to only include upper case letters); limiting the word set (this is common in command type applications, for instance where a user is selecting menu items by writing them with a pen), or constraining the user by making him write the letters in boxes or by making the user write each character in a special way. Unconstrained text entry assumes that the user should be able to write in a very natural way. This supposes that the writer may write on a slant, may form some letters badly, and may use words that are specialist and not found in a custom dictionary the user may also use abbreviations and misspelled words. Most handwriting recognition systems are ‘writer independent’, meaning that the user does not have to train the system to his writing; the term omni-handwriting is applied to such systems and the attraction for these is that the user can quickly use the software without having to go through lengthy enrolment activities similar to those commonly found in continuous speech recognition applications. The process of recognizing the writing is error prone; in the following example, Figure 7, the writing by the child has been captured online, there was no training, the recogniser did not use a dictionary and the recognition happened when the child had finished writing. Figure 7 Writing by a Child The text that the recogniser returned was -Im pushing My Dea up snowdowne crawling on narcis andltness the Judges oxe strict ana sniffy'They send you horne.:ifyou.i sneeze.. From this, some of the errors can be ‘justified’ or explained, e.g. when the a and d in and both became a, it is clearly a result of the ambiguity in the length of the ascender. The division of d in hands into ci is a common error in handwriting recognition, caused by the discrete recognition of the two strokes that made up the d. This is a feature of recognition algorithms that allow discrete writing, and is one reason why cursive writing tends to be better recognised. Where are has become oxe, this can be explained by the breaking of a into an o and a line which then combined with the r stroke to suggest an x. It is likely that the child paused during the construction of a, thus giving the recognition algorithm information to suggest that the line was part of a new character. Because of the erroneous nature of the technology, it common to evaluate handwriting recognition interfaces with regard to the accuracy of the recognition process. Research studies tend to report error rates and accuracy rates that are generally derived from information about what the user wrote and what the recogniser subsequently output. Reported accuracy rates for pen-based input devices vary according to the type of writing that is supported; special alphabets like Graffiti result in recognition rates that are significantly better than that that can be achieved with natural writing (MacKenzie & Zhang, 1997). A study by (MacKenzie & Chang, 1999) tested accuracy rates with 32 subjects copying words of discrete characters onto a tablet, using a constrained grid presented on the screen for the letters to be formed in. Accuracy rates of between 87% and 93% were reported for the two systems used in this way. (Frankish, Hull, & Morgan, 1995) reported accuracy rates for free form text (natural text) that averaged 87%, and rose to 91% when only lower case letters were used. In early studies it was found that non-cursive writing was better recognised than cursive writing (Gibbs, 1993), however later studies seem to suggest that with improved algorithms, cursive writing is better than discrete. Some of the errors that are common have been shown in the description that follows Figure 7 above and the description there indicates that these could be avoided by careful writing. However, there are some errors that can almost never be avoided, known as legitimate errors, and these can account for around 10% of the errors in a recognition-based system. An example of a legitimate error is when ‘o’ is recognised as ‘O’; a mistake that even a human might make (Allen, Hunter, Jacobsen, & Miller, 1994). It should be noted at this point that all of the studies described above have been carried out with adult users, the only reported work on handwriting recognition by children has been our own, The published work in this area includes (Read, MacFarlane, & Casey, 2002a), (Read, MacFarlane, & Casey, 2003a) which focus on the general usability of the technology and its usefulness to support the writing process, and (Read, MacFarlane, & Casey, 2002b), (Read, MacFarlane, & Casey, 2001), (Read, MacFarlane, & Casey, 2003b) which report on how children deal with and recover from errors, how the technology compares to other text input methods, and the mental models that children have of recognition technology have all been reported Can Children Write Stories using Handwriting Recognition? The context of the work The findings that are presented here largely derive from a single classroom study with UK children aged 7 and 8 in which children carried out free writing (composition) in the classroom. An example of the sort of work that was produced is shown in Figure 8 Figure 8 Writing that was produced by one child Introduction The children that took part in this study wrote stories using three different stimuli and three different writing environments. The stimuli were based around parables and followed on from work that they had been doing in their literacy classes. The three writing environments were pen and paper, the word processor and a specially constructed handwriting recognition interface (CobWeb) (Shown in Figure 9) Figure 9 The CobWeb interface The children were given some training in the use of the word processor and the CobWeb interface. The order in which the children came to the interface, and the order in which the different stories were constructed were randomised, see (Read, MacFarlane, & Horton, 2004) for a fuller description of the experimental procedure. Experimental Results Results Children wrote a similar amount of writing at the CobWeb interface as they did at the pen and paper interface, with less writing being done at the keyboard. The class teacher measured the quality of the writing and this was slightly higher for the writing done on pen and paper than for the other two modes. The children were asked which of the interfaces they preferred and this indicated that the keyboard and word processor were the favoured choice (despite being poor for both quality and quantity!) The average error rates for the handwritten text were calculated and were found to be 27.8% with a standard deviation of 15.7%. Discussion – Experimental Design, Usability and Digital Ink for Writing The handwriting recognition interface compared favorably to pen and paper, but there are some problems with it with respect to usability. These can be classified under four headings; Text Input errors Handwriting Errors Habit Driven Errors Hardware Dependent Errors The first ones, user errors that would happen in any text input instance, for example spelling errors can be reduced by the use of phonic spell checking and using software that speaks out the spellings for children. Secondly, there were the errors that were caused by the input mode, ie. Handwriting. These included the incorrect formation of letters (when the stroke order is not the same as the stroke order in the recognition algorithm), the poor construction of individual letters (which may be too slanted, or too open), and the incorrect construction of joins in cursive writing which can all result in poor recognition. The solution for these is either training of the child or the addition of more flexibility into the recognition algorithms. A third category of problems can be termed the habit driven problems. These relate to the natural ways in which children write, inherited from their paper-based experiences and carried forward to the handwriting recognition interface. These included instances when children scribbled over errors that they noted on the screen and when they added things to words that they saw on the screen (commonly making ascenders or descenders longer, closing an incomplete join or inserting a missing character); this ‘seemingly’ natural editing also caused the recognition to go wrong and the solutions for this are rather more complex than simply changing algorithms. It is not especially desirable to force the child to behave in a different way than he or she would, and so the interface, not the child, needs to adapt to this sort of behavior. There is some evidence that the designers of handwriting software are moving towards improved interfaces, and the use of mark up technology and extensions of the delayed t stroke algorithm represent possible solutions. The final set of problems that were noted in this work were hardware dependent, that is, they related to the particular set up of the prototype interface and so were easily remedied! These included the difficulties that children had with the separation of the graphics tablet and the screen (solved by using tablet PC technology – see below), the time lag of the software – that is the time duration set by the software to determine when the child has had the pen off the surface for a long enough time to dictate that a space is required (sometimes this was too fast or too slow), this can be made adaptive to each child’s needs, and finally problems with starting writing in the top left hand corner of the screen which sometimes opened a menu (again easily solved) Discussion of Future Possibilities (Hayes & Flower, 1986) break writing into three overlapping phases, these being ♦ Planning and collecting ♦ Initial drafting leading to more final writing ♦ Revising and editing The first two of these, broadly match the processes A, B and C in Figure 1 above, but the Hayes and Flower model also makes mention of revision and editing. In all phases of the writing process, the use of tablet technology and digital ink is supportive for children writing. In the planning phase, pencil and paper have traditionally been used for note taking, and for storing and recording creative ideas and thoughts. Notes for subsequent writing, sketches, and diagrams can be made almost as easily with pen technology as with paper, especially where there is no requirement for recognition. The newer technology such as the digital pen (Figure 10) offers great promise for the writing classroom of the future. With digital pens, the writing is presented as regular ink to the child and is also stored in the pen. This allows for the selective download of written material to the computer where it can later be manipulated, stored and changed into ASCII text if required. Figure 10 Digital Pen and Paper The transcription phase might favour keyboard input if the child was sufficiently able, particularly since this would result in reasonably reliable ASCII text. However the use of keyboards for input for young writers is not well researched and in particular the effect of them having to break their words into characters has not been well investigated and this may have an effect on their ability to learn spellings. Other aspects of keyboard input that have not been well investigated are the possible detrimental effects on the children’s muscles and posture, the strain on the eyes and the increased cognitive load caused by the child having to continually change his focus from the keyboard to the screen, this leads to errors and increases the cognitive load on the child. With pen input on a tablet PC, like pencil and paper, the child needs only to focus on one thing. Figure 11 Tablet PC The revising and editing phase is undoubtedly facilitated by the use of computer aided text processing. Traditionally, this has been enabled by word processing software manipulating ASCII text, but it may be more beneficial for children to preserve their handwritten work and manipulate this as digital ink. ConclusionThis paper has described some of the possibilities and problems for handwriting recognition with youngwriters. Given the rapid changes in technology, the possibilities for digital ink are worth investigating andcould bring significant changes to the way that children learn to write and the way that they develop theirwriting skills. One observation that is particularly interesting to the writing community is that text that isconstructed at the computer interface can be logged over time, resulting in rich data for research anddevelopment. It is now possible to track the multiple changes that a child (or an adult) may make to apiece of writing.Further work by the authors is concentrating on the effect that digital technology has on the writingprocesses, with respect to quality, quantity and experience. ReferencesAllen, T., Hunter, W., Jacobsen, M., & Miller, M. (1994). Comparing several discrete handwritingrecognition algorithms.: AT & T GIS Human Interface Technology Centre, New User Interface Group.Bearne, E. (1998). Making Progress in English. London: Routledge.Beck, N., & Fetherston, T. (2003). The Effects of Incorporating a Word Processor Into a Year ThreeWriting Program. Information Technology in Childhood Education, Annual, 139 161.Cochran-Smith, M. (1991). Word Processors and Writing in Elementary Classrooms: A Critical Review ofRelated Literature. Review of Educational Research, 6(1), 107 155.Cooper, W. E. (Ed.). (1983). Cognitive Aspects of Skilled Typewriting. New York, USA: Springer Verlag.Day, J. (1994). Is good looking writing good writing?, Computers and Dyslexia.: Computer ResourceCentre.Fisher, E. (1994). Joint composition at the computer: Learning to talk about writing. Computers and Composition, 11, 251 262. Frankish, C., Hull, R., & Morgan, P. (1995). Recognition Accuracy and User Acceptance of PenInterfaces. Paper presented at the CHI'95, Denver.Gibbs, M. (1993). Handwriting Recognition: a comprehensive comparison. Pen, 31 35.Hayes, J. R., & Flower, L. S. (1986). Writing research and the writer. Amercan Psychologist, 41(10),1106 1113.Hult, C. (1986). The computer and the inexperienced writer. Paper presented at the College Compositionand Communication, New Orleans.Jones, L., & Pellegrini, A. D. (1996). The effects of social relationships, writing media, and microgeneticdevelopment on first grade students written narratives. American Educational Research Journal, 33, 691718.MacKenzie, I. S., & Chang, L. (1999). A performance comparison of two handwriting recognizers.Interacting with Computers, 11(3), 283 297.MacKenzie, I. S., & Zhang, S. X. (1997). The immediate usability of Graffiti. Paper presented at the TheGraphics Interface, Toronto.McHale, K., & Cermak, S. (1992). Fine motor activities in elementary school: Preliminary findings andprovivional implications for children with fine motor problems. American Journal of Occupational Therapy,46(10), 898 903.Monahan, B. (1984). Revision strategies of basic and competent writers as they write for differentaudiences. Research in the teaching of English, 18, 288 304.Moseley, D., Higgins, S., Bramald, R., Hardman, F., Miller, J., Mroz, M., Tse, H., Newton, D., Thompson,I., Williamson, J., Halligan, J., & Bramald, P. (1999). Ways forward with ICT: Effective Pedagogy using Information and Communication Technology for Literacy and Numeracy in Primary Schools. Newcastle:Newcastle University.Nicholson, R. I. (1999). Reading Skill and Dyslexia. In D. J. Messar & S. Millar (Eds.), Exploring Developmental Psychology. London: Arnold.Norman, D. A., & Fisher, D. (1982). Why alphabetic keyboards are not easy to use: Keyboard layoutdoesn't much matter. Human Factors, 24, 509 515.Papert, S. (1980). Mindstorms, Children, Computers and Powerful Ideas. Great Britain: Basic Books.Peters, M. L. (1985). Spelling: Caught or Taught: A New Look. London: Routledge and Kegan Paul.Plamondon, R., & Srihari, S. N. (2000). On-line and Off-Line Handwriting Recognition: A ComprehensiveSurvey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(1), 63 84.Read, J. C., MacFarlane, S. J., & Casey, C. (2001). Measuring the Usability of Text Input Methods for Children. Paper presented at the HCI2001, Lille, France.Read, J. C., MacFarlane, S. J., & Casey, C. (2002a). Designing a Handwriting Recognition Based Writing Environment for Children. Paper presented at the 8th International EARLI SIG Writing Conference,Staffordshire, England.Read, J. C., MacFarlane, S. J., & Casey, C. (2002b). Oops! Silly me! Errors in a HandwritingRecognition-based Text entry Interface for Children. Paper presented at the NordiChi 2002, Aarhus,Denmark.Read, J. C., MacFarlane, S. J., & Casey, C. (2003a). 'Good enough for what? Acceptance of Handwriting Recognition Errors by Child Users. Paper presented at the IDC 2003, Preston, UK.Read, J. C., MacFarlane, S. J., & Casey, C. (2003b). What's going on? Discovering what Childrenunderstand about Handwriting Recognition Interfaces. Paper presented at the IDC 2003, Preston, UK.Read, J. C., MacFarlane, S. J., & Horton, M. (2004). The Usability of Handwriting Recognition for Writingin the Primary Classroom. Paper presented at the HCI 2004, Leeds, UK.Sassoon, R. (1990). Handwriting: A New Perspective. Leckhampton: Stanley Thornes. Srihari, S. N., & Srihari, R. K. (1997). Written Language Input. In R. Cole & J. Mariani & H. Uskoreit & A.Zaenen & V. Zue (Eds.), Survey of the state of the art in human language technology. Pisa: CambridgeUniversity Press.Swanson, H. L., & Berninger, V. W. (1996). Individual Differences in Children's Working Memory andWriting Skill. Journal of Experimental Child Psychology, 63(2), 358 385.Zbikowski, J. M., & Pan, A. C. (1995). A model of learning to teach writing with computers. In D. Carey &R. Carey & D. A. Willis & J. Willis (Eds.), Technology and Teacher Education Annual 1994 (pp. 118 -121). Charlottesville, VA: Association for the Advancement of Computing in Education (AACE).